The present invention is generally directed towards stereoscopic image synthesis and more particularly toward a method of converting two dimensional (2D) images for further encoding, transmission and decoding for the purpose of stereoscopic image display on two dimensional (2D) or three dimensional (3D) stereoscopic displays systems.
Recent improvements in technology in the areas of compact high performance video projection systems, image processing, digital video and liquid crystal panels have made possible many practical 3D display systems utilising both active and passive polarising glasses and both single and multi-viewer autostereoscopic displays.
Three dimensional display systems have moved out of the arena of being technological curiosities and are now becoming practical display systems for entertainment, commercial and scientific applications. There has now emerged the requirement for 3D media to display on these devices. Traditionally there has been just two ways to produce this 3D media (ie media produced that contains image information for at least two separate views of the same scene from different perspective""s). These are:
1) Generation of two separate views (usually in real time) by a computer.
2) Videoing or filming with two laterally displaced cameras.
In the case of computer generated images for usage in Computer Aided Design (CAD) systems, simulators or video game equipment, it is not a complex process to produce two separate images with different perspective""s.
The filming of movies utilising two laterally displaced cameras to produce 3D has been well understood for many years. However, there are many problems with this approach. It is considerably more difficult to film or video in 3D than in 2D because there are limits to the permissible distance between the nearest and farthest objects in the scene (practical 3D depth of field) as well as framing problems (such as near objects being seen on only one camera and thus highlighting the inaccuracy of the 3D image generation when re-played. Another problem is maintaining a smooth pan without causing false 3D artefacts due to latency between the images from the two cameras and so on.
Because of the complexity, high cost of production and implementation, and the fact that there are as yet still only a very small number of 3D display systems being produced for the domestic and commercial markets there has not been a large incentive for the major producers of films or videos to produce 3D media. However, if a technique was devised that would allow conventional 2D films to be re-processed into a 3D version then it would be possible to not only convert new films into 3D format for significantly less cost than filming them directly in 3D in the first place but it would also make possible the re-processing of the vast archives of 2D film and video material for re-release to both the cinema and video markets.
It would however be advantageous to be able to convert an existing 2D image so that it can be viewed as a 3D image. One way of achieving this is to convert a single 2D image to two separate left and right images by a xe2x80x98cut and pastexe2x80x99 technique. In this technique, an object is xe2x80x98cutxe2x80x99 from the image and laterally displaced left or right then xe2x80x98pastedxe2x80x99 back onto the original image to produce the required separate images. This however results in a blank region in the area formally occupied by the object within the image.
It is therefore an object of the present invention to overcome or minimise at least one of these problems.
With this in mind, the present invention provides in one aspect, a method of producing left and right eye images for a stereoscopic display from an original 2D image, wherein selected areas of said original image are displaced by a determined amount and direction to thereby generate stretched images, said stretched images forming said left and right eye images.
The two converted images when respectively viewed by the left and right eye of a viewer can provide a 3D image without any blank region as would be the case with images produced by the xe2x80x98cut and pastexe2x80x99 technique.
This document covers some of the major algorithmic processes involved in the conversion of 2D media to 3D format, a new composite data format which is suitable for the communication and storage of this new 3D media. It also discusses several hardware implementations for the real-time encoding, transmission and decoding of this new 3D format.
The main benefits of the technology is that there are significant cost saving and media supply advantages.ie. Only one camera has to be used for filming. Due to the nature of the 2D to 3D conversion process it is possible to package and transfer the image media virtually unchanged except for the addition of a small packet of 3D data that doesn""t in any way encumber the final 2D display process. In fact it enables the images to be displayed on standard 2D televisions in 2D or 3D (usage of shutter glasses or similar required) without degradation of image quality whilst also allowing display on 3D televisions or other displays in 3D.
The final stage of the 2D to 3D conversion process is completed in real-time at the receiver and thus the increased bandwidth requirements for displaying 3D images is local to the television decoder and does not adversely affect the channel handling capacity of the television carrier.
According to another aspect of the present invention there is provided a method of describing the changes to be made to an original 2D image for converting said 2D image into stretched images for a stereoscopic display.
According to a further aspect of the present invention, there is provided a method of encoding a video signal of a 2D image for allowing conversion of said video 2D image to stretched images for a stereoscopic display.
According to yet another aspect of the present invention, there is provided a method of receiving a video signal of a 2D image including encoding data, and extracting the encoding data from the video signal for allowing conversion of the 2D image to stretched images for a stereoscopic display.
According to a further aspect of the present invention, there is provided a method of manipulating a 2D video image with encoding data to thereby provide stretched images for a stereoscopic display.
According to another aspect of the present invention, there is provided a method of producing left and right eye images for a stereoscopic display from an original 2D image including the steps of:
a) identifying at least one object within said original image,
b) outlining said or each object,
c) defining a depth characteristic for said or each object,
d) respectively displacing selected areas of said or each image by a determined amount in a lateral direction as a function of the depth characteristic of said or each object, to form two stretched images for viewing by the left and right eyes of the viewer.
These image pairs may be either mirrored or similar to each other so that the stereoscopic 3D effect is optimised.
The image may include a plurality of objects with each object being provided with a said respective depth characteristic. Images may be converted on an individual basis. Alternatively, a series of related images as in a video or film may be converted.
The image may be digitised and the image may be stretched or converted electronically by temporarily placing a mesh over the image, the mesh initially having a plurality of parallel lateral mesh lines and a plurality of parallel longitudinal mesh lines positioned at right angles to the lateral mesh lines. Each intersection of the mesh lines on the mesh may provide a mesh sub-point. The image can move together with the mesh so that distortion of the mesh results in the stretching of the underlying image. The mesh lines may remain continuous to provide for a smooth stretching of the image. The amount of displacement of each of the mesh sub-points from their initial position may provide the conversion data for said original image. The sub-points may be displaced in a lateral direction.
The displacement of the mesh sub-points may also be defined by a mathematical algorithm to thereby provide for automatic conversion of images. Further enhancements to the method could be to add shadow, blurring and motion interpolation data to the conversion data including force paralex information and field delay and direction for motion paralex delays.
It would be advantageous to be able to use existing image transmission systems to transmit an image that can be viewed as a 3D image. The present invention can be applicable for use in image transmission systems sending video signals that provide 2D images.
According to a further aspect of the present invention there is provided a method of producing a set of xe2x80x98object scriptingxe2x80x99 data from the conversion/stretching process that describes which objects in an image are selected for processing, how they will be processed, their priority or otherwise over other objects and their depth characteristics. This scripting data may be stored in the memory of a computer for later usage to reprocess the original 2D images or transmitted to another site (assuming the other site has the same 2D images) for reproduction of the 3D images.
Therefore, according to a further aspect of the present invention there is provided an encoder for encoding a video signal providing a 2D video image, including:
adding conversion data to the video signal to provide the encoded signal, the data defining the displacement of respective selected points of the video image for converting the video image to a stretched image for a stereoscopic display.
By adding the conversion data to the video signal, existing transmission systems can be used to transmit the encoded signal. Various arrangements may be provided to add the conversion data to the video signal. For example, the data may be included in the blank lines of the transmitted video image at the lop and bottom of the image or in the horizontal sync period or horizontal overscan regions of each line.
This invention is not limited to the conversion of existing 2D video images. Rather the process can be readily used lo produce the conversion data simultaneously with the creation of the 2D video image.
Therefore, according to a further aspect of the present invention there is provided a method of producing a 2D video image encoded with 3D conversion data including:
capturing video images from a plurality of video cameras;
comparing the video images from each respective video camera in order to produce the conversion data, the conversion data defining the displacement of respective points of the video image for converting the video image to a stretched image for a stereoscopic display; and
combining the video signal from one of said video cameras with the conversion data to thereby produce the encoded video signal.
In a further aspect of the present invention there is provided a method of producing a 2D video image encoded with 3D conversion data including:
capturing left and right eye video images from a stereoscopic video camera;
comparing the left and right eye video images from the stereoscopic video camera in order to produce the conversion data, the conversion data defining the displacement of respective points of the video image for converting the video image to a stretched image for a stereoscopic display; and
combining the video signal from said video camera with the conversion data to thereby produce the encoded video signal.
In yet a further aspect of the present invention there is provided a system of producing a 2D video signal encoded with 3D conversion data including:
at least a first and second video camera displaced laterally relative to each other;
conversion means for producing the conversion data, said conversion means receiving data from said video cameras and comparing the data to thereby produce the conversion data, said conversion data defining the displacement of respective points of the video image from one of said video cameras for converting the video image to a stretched image for a stereoscopic display;
an encoder means for combining the video signal from said one video camera with the conversion data from said conversion means to thereby produce the encoded video signal.
Where the 2D video image encoded with the 3D conversion data is only required for a single viewer, only two video cameras are required, each camera representing the view seen by the left and right eyes of the viewer.
In yet another aspect the present invention provides a system of producing a 2D video signal encoded with 3D conversion data including:
a stereoscopic video camera;
conversion means for producing the conversion data, said conversion means receiving data from said video camera and comparing the data to thereby produce the conversion data, said conversion data defining the displacement of respective points of the video image for converting the video image to a stretched image for a stereoscopic display;
an encoder means for combining the video signal with the conversion data from said conversion means to thereby produce the encoded video signal.
According to yet another aspect of the present invention, there is provided a decoder for decoding a video signal for providing a stereoscopic display, the signal providing a 2D video image and further including conversion data for converting the video image, the conversion data defining the displacement of respective points of the video image for converting the video image to a stretched image for a stereoscopic display, the decoder including:
a) means for receiving the video signal; and
b) decoding means for reading the conversion data and for controlling the video signal to thereby provide a converted video signal.
The decoder may include
a) an RGB or component video converter for converting the video signal Into separate video components thereof,
b) analog to digital conversion means for converting each video component to a respective digital signal, and
c) digital storage means for storing said digital signals.
The decoding means may control a variable frequency clock means controlling the read out rate of the digital storage means whereby the storage means are read out at a variable rate. This results in the video image being stretched or compressed in accordance with the conversion data.
Alternatively, the RGB or video components may be read into the storage means at a variable rate, and read out of the storage means at a fixed rate.
The decoder may process a single video line or may also process multiple lines such as in a complete field or frame. In this case the full mesh from the conversion data is restored with pixel distortions (lateral shifts) being calculated over the complete field or frame.
The storage means may be in the form of a dual port RAM line store.
A digital to analog conversion means may be provided for converting the read out digital signal to a converted video signal to enable viewing on viewing means. The viewing means may include a television or other screen for viewing the converted video image. The viewing means may further include shutter glasses controlled by the decoder to enable the converted video image to be viewed as a stereoscopic image.
Alternatively, the decoder may include parallel storage means for storing digital signal for the converted left and right video images respectively. The viewing means may then include a display unit for simultaneously projecting the left and right video images.
The decoder means may include separation means for separating the conversion data from the video signal.
According to a further aspect of the present invention, there is provided a stereoscopic image display system including:
a) an encoder for encoding a video signal providing a video image with conversion data, the conversion data defining displacement of respective points of the video image for converting the video image to a stretched image for a stereoscopic display; and
b) a decoder for separating the conversion data from the video signal and for converting the video signal as a function of the conversion data.
According to yet another aspect of the present invention, there is provided a multiviewer stereoscopic display system including:
a) a decoder for decoding a video signal for providing a stereoscopic display, the signal providing a 2D video image and further including conversion data for converting the video image, the conversion data defining the displacement of respective points of the video image for converting the video image to a stretched image for a stereoscopic display, the decoder including means for receiving the video signal and decoding means for reading the conversion data and for controlling the video signal to thereby provide a converted video signal;
It will be convenient to further describe the invention by reference to the accompanying drawings which illustrate possible implementations of the present invention.
Other implementations of the invention are also possible and consequent the particularities of the accompanying drawings is not to be understood as superseding the generality of the preceding description.